Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text

Long unpunctuated texts containing complex linguistic sentences are a stumbling block to processing any low-resource languages. Thus, approaches that attempt segment lengthy with no proper punctuation into simple candidate vitally important preprocessing task in many hard-to-solve NLP applications. To this end, we propose solution for segmenting Arabic potentially independent clauses. This consists of: (1) detection model built on top of multilingual BERT-based model, and (2) some generic rules validating the resulting segmentation. Furthermore, optimize strategy applying these using our suggested greedy-like algorithm. We call proposed PDTS (standing Punctuation Detector Text Segmentation). Concerning evaluation, showcase how can be effectively employed as text tokenizer documents (i.e., mimicking transcribed audio-to-text documents). Experimental findings across two evaluation protocols (involving an ablation study human-based judgment) demonstrate is practically effective both performance quality computational cost. In particular, reach average F-Measure score approximately 75%, indicating minimum improvement roughly 13% compared state-of-the-art competitor models).